System And Methods For Performing Surgery On A Patient At A Target Site Defined By A Virtual Object

ABSTRACT

System and methods for performing surgery at a target site defined by a virtual object. A surgical navigation system includes a patient tracker to be attached to a patient. A localizer cooperates with the patient tracker and generates localizer data associated with the target site during the surgery. The surgical navigation system also includes a vision device to generate image data associated with the target site and surfaces surrounding the target site. A navigation computer in communication with the localizer and the vision device is configured to determine a region to be avoided outside of the target site based on the localizer data and the image data. In some cases, a second virtual object is generated to define the region to be avoided so that a surgical instrument used during the surgery avoids the region.

RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 62/273,543, filed Dec. 31, 2015, the entirecontents and disclosure of which are hereby incorporated by referenceherein.

TECHNICAL FIELD

The present disclosure relates generally to a system and method forperforming surgery on a patient at a target site defined by a virtualobject.

BACKGROUND

Navigation systems assist users in precisely locating objects. Forinstance, navigation systems are used in industrial, aerospace, andmedical applications. In the medical field, navigation systems assistsurgeons in precisely placing surgical instruments relative to a targetsite in a patient. The target site usually requires some form oftreatment, such as tissue removal. In some cases, the target site isdefined in the navigation system using a virtual object, such as a 3-Dmodel. A representation of the virtual object can be displayed to theuser during surgery to assist the user in visualizing placement of atreatment end of the instrument relative to the target site. Forinstance, the target site may be associated with a bone of the patientand the virtual object may define a volume of the bone to be removed bythe treatment end of the instrument.

Conventional navigation systems employ a localizer that cooperates withtrackers to provide position and/or orientation data associated with theinstrument and the target site, e.g., the volume of the bone to beremoved. The localizer is usually placed so that it has a field of viewof the trackers. The trackers are fixed to the instrument and to thepatient to move in concert with the instrument and the patient. Thetracker attached to the patient is attached to the bone being treatedthereby maintaining a rigid relationship with respect to the target siteowing to the rigid nature of the bone. By using separate trackers on theinstrument and the patient, the treatment end of the instrument can beprecisely positioned to stay within the target site.

Often, the target site is located adjacent to sensitive anatomicalstructures, such as soft tissue, that should be avoided during surgery.These sensitive anatomical structures are difficult to track usingconventional trackers, as these sensitive anatomical structures canshift relative to the trackers due to their elastic and/or flexiblenature. Just as often, retractors or other tools are located near thetarget site that should also be avoided during the surgery. Theretractors or other tools could be tracked in the same manner as theinstrument being used for treating the patient, but adding trackers tothe retractors and other tools can substantially increase costs andcomplexity in the navigation system, particularly by increasing thenumber of objects to be tracked by the navigation system. As a result,in current surgical procedures, avoidance is sometimes theresponsibility of the user, so extreme care must be taken by the user toavoid sensitive anatomical structures and untracked tools that may benear the target site.

Thus, there is a need in the art for navigation systems and methods thataddress the identification of sensitive anatomical structures and/orother structures that are to be avoided during surgery.

SUMMARY

In one embodiment, a surgical navigation system for performing surgeryat a target site defined by a virtual object is provided. The surgicalnavigation system includes a patient tracker to be attached to apatient. A localizer cooperates with the patient tracker and generateslocalizer data associated with the target site during the surgery. Thesurgical navigation system also includes a vision device to generateimage data associated with the target site and surfaces surrounding thetarget site. A navigation computer in communication with the localizerand the vision device is configured to determine a region to be avoidedoutside of the target site based on the localizer data and the imagedata.

In another embodiment, a robotic surgical system for performing surgeryat a target site defined by a virtual object is provided. The roboticsurgical system includes a robotic device. An end effector is coupled tothe robotic device for treating the target site. The robotic surgicalsystem also includes a patient tracker to be attached to a patient. Alocalizer cooperates with the patient tracker and generates localizerdata associated with the target site during the surgery. The roboticsurgical system includes a vision device to generate image dataassociated with the target site and surfaces surrounding the targetsite. A navigation computer in communication with the localizer and thevision device is configured to determine a region to be avoided outsideof the target site during the surgery based on the localizer data andthe image data. The navigation computer is in communication with therobotic device so that the robotic device is operable to move the endeffector with respect to the target site while avoiding the region to beavoided.

In another embodiment, a method of performing surgery at a target sitedefined by the virtual object is provided. The method comprisesgenerating localizer data associated with the target site while apatient tracker is attached to a patient. Image data associated with thetarget site and surfaces surrounding the target site is also generated.The method further comprises determining a region to be avoided outsideof the target site during the surgery based on the localizer data andthe image data.

These systems and methods provide several advantages. For instance, bycapturing both localizer data using the localizer and image data usingthe vision device, the navigation computer is able to identify theregion to be avoided that is located outside of the target site. As aresult, these systems and methods, in some embodiments, provide foraccurate placement of surgical instruments to avoid sensitive anatomicalstructures that are otherwise difficult to track and to avoid othertools near the target site that may not be outfitted with separatetrackers.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages will be readily appreciated as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings.

FIG. 1 is a perspective view of a robotic surgical system comprising alocalizer and a vision device.

FIG. 2 is a schematic view of a control system for controlling therobotic surgical system.

FIG. 3 is a perspective view of coordinate systems used in the roboticsurgical system.

FIG. 4 is an illustration of image data from the vision device beingcombined with localizer data from the localizer to yield a virtualobject defining a region to be avoided.

FIG. 5 is a flow chart of steps carried out by a method of treating thepatient.

DETAILED DESCRIPTION

As shown in FIG. 1, a system 20 for treating a patient 22 isillustrated. The system 20 is shown in a surgical setting such as anoperating room of a medical facility. In the embodiment shown, thesystem 20 comprises a machining station 24 and a guidance station 26.The guidance station 26 is set up to track movement of various objectsin the operating room. Such objects include, for example, a surgicalinstrument 30, a femur F of a patient, and a tibia T of the patient. Theguidance station 26 tracks these objects for purposes of displayingtheir relative positions and orientations to a user and, in some cases,for purposes of controlling or constraining movement of the surgicalinstrument 30 relative to target sites. The surgical instrument 30 isshown as part of the machining station 24. However, in otherembodiments, the surgical instrument 30 is manually held and moved bythe user.

The target sites to be treated by the surgical instrument 30 are definedby virtual objects. In the embodiment shown, a femur target site TS isshown, which is associated with the femur F. Of course, several othertarget sites, such as a target site for the tibia T, are also possible,with each being defined by its own separate virtual object. The virtualobjects representing the target sites are pre-operatively set by theuser and/or automatically generated to define volumes of material to betreated, trajectories for the surgical instrument 30, planes to be cutby the surgical instrument 30, bores to be drilled, and the like. In theembodiment shown, a virtual object VB (see FIG. 4) defines the volume ofmaterial to be removed from the femur F. In some cases, the virtualobjects are set or re-set intraoperatively, i.e., during the surgicalprocedure. It should be appreciated that although the description setforth herein relates to orthopedic surgical procedures, the systems andmethods described herein are likewise suitable for any type of surgicalprocedure.

The guidance station 26 includes a navigation cart assembly 32 thathouses a navigation computer 34. A navigation interface is in operativecommunication with the navigation computer 34. The navigation interfaceincludes a first display 36 adapted to be situated outside of thesterile field and a second display 38 adapted to be situated inside thesterile field. The displays 36, 38 are adjustably mounted to thenavigation cart assembly 32. First and second input devices 40, 42 suchas a keyboard and mouse can be used to input information into thenavigation computer 34 or otherwise select/control certain aspects ofthe navigation computer 34. Other input devices are contemplatedincluding a touch screen (not shown) or voice-activation.

A localizer 44 communicates with the navigation computer 34. In theembodiment shown, the localizer 44 is an optical localizer and includesa localizer camera unit 46. The localizer camera unit 46 has an outercasing 48 that houses one or more optical position sensors 50. In someembodiments at least two optical sensors 50 are employed, preferablythree, four, or more. The optical sensors 50 may be three separatecharge-coupled devices (CCD). In one embodiment three, one-dimensionalCCDs are employed. It should be appreciated that in other embodiments,separate localizer camera units, each with a separate CCD, or two ormore CCDs, could also be arranged around the operating room. The CCDsdetect infrared signals. Additionally, the localizer 44 may employdifferent modalities and may be an electromagnetic localizer, RFlocalizer, ultrasound localizer, or any other conventional localizercapable of tracking objects.

The localizer camera unit 46 is mounted to an adjustable arm to positionthe optical sensors 50 with a field of view of the below discussedtrackers that, ideally, is free from obstructions. In some embodimentsthe localizer camera unit 46 is adjustable in at least one degree offreedom by rotating about a rotational joint. In other embodiments, thelocalizer camera unit 46 is adjustable about two or more degrees offreedom.

The localizer camera unit 46 includes a localizer camera controller 52in communication with the optical sensors 50 to receive signals from theoptical sensors 50. The localizer camera controller 52 communicates withthe navigation computer 34 through either a wired or wireless connection(not shown). One such connection may be an IEEE 1394 interface, which isa serial bus interface standard for high-speed communications andisochronous real-time data transfer. The connections could also use acompany specific protocol. In other embodiments, the optical sensors 50communicate directly with the navigation computer 34.

Position and orientation signals and/or data are transmitted to thenavigation computer 34 for purposes of tracking objects. The navigationcart assembly 32, displays 36, 38, and localizer camera unit 46 may belike those described in U.S. Pat. No. 7,725,162 to Malackowski, et al.issued on May 25, 2010, entitled “Surgery System,” hereby incorporatedby reference.

Navigation computer 34 has the displays 36, 38, central processing unit(CPU) and/or other processors 62, memory (not shown), and storage (notshown) necessary for carrying out the functions described herein. Thenavigation computer 34 is loaded with software as described below. Thesoftware converts the signals received from the localizer camera unit 46into localizer data representative of the position and orientation ofthe objects being tracked.

Guidance station 26 is operable with a plurality of tracking devices 54,56, 58, also referred to herein as trackers. In the illustratedembodiment, one tracker is 54 is firmly affixed to the femur F of thepatient and another tracker 56 is firmly affixed to the tibia T of thepatient. Trackers 54, 56 are firmly affixed to sections of bone.Trackers 54, 56 may be attached to the femur F and tibia T in the mannershown in U.S. Pat. No. 7,725,162, hereby incorporated by references.Trackers 54, 56 could also be mounted like those shown in U.S. PatentApplication Publication No. 2014/0200621, filed on Jan. 16, 2014,entitled, “Navigation Systems and Methods for Indicating and ReducingLine-of-Sight Errors,” hereby incorporated by reference herein. In yetfurther embodiments, the trackers 54, 56 could be mounted to othertissues of the anatomy.

An instrument tracker 58 is firmly attached to the surgical instrument30. The instrument tracker 58 may be integrated into the surgicalinstrument 30 during manufacture or may be separately mounted to thesurgical instrument 30 in preparation for surgical procedures. Atreatment end of the surgical instrument 30, which is being tracked byvirtue of the instrument tracker 58, may be a rotating bur, electricalablation device, or the like.

The trackers 54, 56, 58 can be battery powered with an internal batteryor may have leads to receive power through the navigation computer 34,which, like the localizer camera unit 46, preferably receives externalpower.

In the embodiment shown, the surgical instrument 30 is attached to amanipulator 66 of the machining station 24. The manipulator 66 may alsobe referred to as a robotic device or a robotic arm. Such an arrangementis shown in U.S. Pat. No. 9,119,655, entitled, “Surgical ManipulatorCapable of Controlling a Surgical Instrument in Multiple Modes,” thedisclosure of which is hereby incorporated by reference. It should beappreciated that in other embodiments, the surgical instrument 30 ismanually manipulated without any robotic constraint on its positionand/or orientation. The surgical instrument 30 may be any surgicalinstrument (also referred to as a tool) that is useful in performingmedical/surgical procedures. The surgical instrument 30 may be a burringinstrument, an electrosurgical instrument, an ultrasonic instrument, areamer, an impactor, a sagittal saw, or other instrument. In someembodiments, multiple surgical instruments are employed to treat thepatient, with each being separately tracked by the localizer 44.

The optical sensors 50 of the localizer 44 receive light signals fromthe trackers 54, 56, 58. In the illustrated embodiment, the trackers 54,56, 58 are active trackers. In this embodiment, each tracker 54, 56, 58has at least three active tracking elements or markers for transmittinglight signals to the optical sensors 50. The active markers can be, forexample, light emitting diodes or LEDs 60 transmitting light, such asinfrared light. The optical sensors 50 preferably have sampling rates of100 Hz or more, more preferably 300 Hz or more, and most preferably 500Hz or more. In some embodiments, the optical sensors 50 have samplingrates of 8000 Hz. The sampling rate is the rate at which the opticalsensors 50 receive light signals from sequentially fired LEDs 60. Insome embodiments, the light signals from the LEDs 60 are fired atdifferent rates for each tracker 54, 56, 58.

Referring to FIG. 2, each of the LEDs 60 are connected to a trackercontroller 61 located in a housing of the associated tracker 54, 56, 58that transmits/receives data to/from the navigation computer 34. In oneembodiment, the tracker controllers 61 transmit data on the order ofseveral Megabytes/second through wired connections with the navigationcomputer 34. In other embodiments, a wireless connection may be used. Inthese embodiments, the navigation computer 34 has a transceiver (notshown) to receive data from the tracker controller.

In other embodiments, the trackers 54, 56, 58 may have passive markers(not shown), such as reflectors that reflect light emitted from thelocalizer camera unit 46. The reflected light is then received by theoptical sensors 50. Active and passive arrangements are well known inthe art.

In some embodiments, the trackers 54, 56, 58 also include a gyroscopesensor and accelerometer, such as the trackers shown in U.S. Pat. No.9,008,757 to Wu, issued on Apr. 14, 2015, entitled “Navigation SystemIncluding Optical and Non-Optical Sensors,” hereby incorporated byreference.

The navigation computer 34 includes the navigation processor 62. Itshould be understood that the navigation processor 62 could include oneor more processors to control operation of the navigation computer 34.The processors can be any type of microprocessor or multi-processorsystem. The term processor is not intended to limit the scope of anyembodiment to a single processor.

The localizer camera unit 46 receives optical signals from the LEDs 60of the trackers 54, 56, 58 and outputs to the navigation processor 62signals relating to the position of the LEDs 60 of the trackers 54, 56,58 relative to the localizer 44. Based on the received optical (andnon-optical signals in some embodiments), navigation processor 62generates data indicating the relative positions and orientations of thetrackers 54, 56, 58 relative to the localizer 44, such as through knowntriangulation methods. In some embodiments, the data is generated by thelocalizer camera controller 52 and then transmitted to the navigationcomputer 34.

Prior to the start of the surgical procedure, additional data are loadedinto the navigation processor 62. Based on the position and orientationof the trackers 54, 56, 58 and the previously loaded data, navigationprocessor 62 determines the position of the treatment end of thesurgical instrument 30 (e.g., the centroid of a surgical bur) and theorientation of the surgical instrument 30 relative to the target sitesagainst which the treatment end is to be applied, such as the femurtarget site TS. In some embodiments, navigation processor 62 forwardsthese data to a manipulator controller 64. The manipulator controller 64can then use the data to control the manipulator 66 as described in U.S.Pat. No. 9,119,655, entitled, “Surgical Manipulator Capable ofControlling a Surgical Instrument in Multiple Modes,” the disclosure ofwhich is hereby incorporated by reference. In one embodiment, themanipulator 66 is controlled with respect to the virtual objects set bythe surgeon. In the embodiment described herein, the virtual object VBdefines the volume of material of the femur F to be removed by thesurgical instrument 30. Thus, the virtual object VB provides a virtualboundary for the treatment end of the surgical instrument 30 to staywithin (i.e., for a separate virtual object associated with thetreatment end of the surgical instrument to stay within).

The navigation processor 62 also generates image signals that indicatethe relative position of the treatment end to the target sites. Theseimage signals are applied to the displays 36, 38. Displays 36, 38, basedon these signals, generate images that allow the surgeon and staff tovirtually view the relative position of the treatment end to the targetsites. In most cases, the images illustrate the treatment end withrespect to one target site at a time. For instance, in a surgicalprocedure in which the femur F and the tibia T are both being treated,the femur target site TS and the relative position of the treatment endof the surgical instrument 30 to the femur target site TS may bevisually represented while material is being removed from the femur F.Likewise, when the user is finished removing material from the femur Fand is ready to remove material from the tibia T, the display 36, 38 mayonly illustrate placement of the treatment end of the surgicalinstrument 30 with respect to the target site associated with the tibiaT.

Referring to FIG. 3, tracking of objects is generally conducted withreference to a localizer coordinate system LCLZ. The localizercoordinate system LCLZ has an origin and an orientation (a set of x, y,and z axes). During the procedure one goal is to keep the localizercoordinate system LCLZ in a known position. An accelerometer (not shown)mounted to the localizer camera unit 46 may be used to track sudden orunexpected movement of the localizer coordinate system LCLZ, as mayoccur when the localizer camera unit 46 is inadvertently bumped bysurgical personnel.

Each tracker 54, 56, 58, and object being tracked also has its owncoordinate system separate from the localizer coordinate system LCLZ.For instance, the trackers 54, 56, 58 have bone tracker coordinatesystem BTRK1, bone tracker coordinate system BTRK2, and instrumenttracker coordinate system TLTR.

In the embodiment shown, the guidance station 26 monitors the positionsof the femur F and tibia T of the patient by monitoring the position ofbone trackers 54, 56 firmly attached to bone. Femur coordinate system isFBONE and tibia coordinate system is TBONE, which are the coordinatesystems of the bones to which the bone trackers 54, 56 are firmlyattached.

Prior to the start of the procedure, pre-operative images of the anatomyof interest are generated, such as pre-operative images of the femur Fand tibia T (or of other tissues or structures in other embodiments).These images may be based on MRI scans, radiological scans or computedtomography (CT) scans of the patient's anatomy. These images are used todevelop virtual models of anatomy of interest, such as virtual models ofthe femur F and tibia T and/or other anatomy to be treated by thesurgical instrument 30. Often the virtual models are 3-D models thatcomprise data representing the entire anatomy being treated or at leasta portion of the anatomy to be treated and data representing the virtualobjects that define the target sites. In the embodiment shown, a virtualmodel VM of the femur is a 3-D model comprising model data thatrepresents a portion of the femur F and the virtual object VB (see FIG.4). The virtual object VB defines the target site TS and the volume ofmaterial to be removed from the femur F during the surgical procedure.The virtual objects may be defined within the virtual models and may berepresented as mesh surfaces, constructive solid geometries (CSG),voxels, or using other virtual object representation techniques.

The pre-operative images and/or the virtual models are mapped to thefemur coordinate system FBONE and tibia coordinate system TBONE usingwell known methods in the art. These pre-operative images and/or virtualmodels are fixed in the femur coordinate system FBONE and tibiacoordinate system TBONE. As an alternative to taking pre-operativeimages, plans for treatment can be developed in the operating room fromkinematic studies, bone tracing, and other methods. These same methodscould also be used to generate the 3-D virtual models previouslydescribed.

During an initial phase of the procedure described herein, the bonetrackers 54, 56 are firmly affixed to the bones of the patient. The pose(position and orientation) of coordinate systems FBONE and TBONE aremapped to coordinate systems BTRK1 and BTRK2, respectively. In oneembodiment, a pointer instrument P (see FIG. 1), such as disclosed inU.S. Pat. No. 7,725,162 to Malackowski, et al., hereby incorporated byreference, having its own tracker PT (see FIG. 1), may be used toregister the femur coordinate system FBONE and tibia coordinate systemTBONE to the bone tracker coordinate systems BTRK1 and BTRK2,respectively. Given the fixed relationship between the bones and theirtrackers 54, 56, positions and orientations of the femur F and tibia Tin the femur coordinate system FBONE and tibia coordinate system TBONEcan be transformed to the bone tracker coordinate systems BTRK1 andBTRK2 so the localizer camera unit 46 is able to track the femur F andtibia T by tracking the trackers 54, 56. These pose-describing data arestored in memory integral with both the manipulator controller 64 andthe navigation processor 62.

The treatment end of the surgical instrument 30 (also referred to as adistal end of an energy applicator) has its own coordinate system EAPP.The origin of the coordinate system EAPP may represent a centroid of asurgical cutting bur, for example. The pose of coordinate system EAPP isfixed to the pose of instrument tracker coordinate system TLTR beforethe procedure begins. Accordingly, the poses of these coordinate systemsEAPP, TLTR relative to each other are determined. The pose-describingdata are stored in memory integral with manipulator controller 64 andnavigation processor 62.

Referring to FIG. 2, a localization engine 100 is a software module thatcan be considered part of the navigation computer 34. Components of thelocalization engine 100 run on navigation processor 62. The localizationengine 100 may run on the manipulator controller 64 and/or thenavigation processor 62.

Localization engine 100 receives as inputs the optically-based signalsfrom the localizer camera controller 52 and, in some embodiments, thenon-optically based signals from the tracker controller (not shown).Based on these signals, localization engine 100 determines the pose ofthe bone tracker coordinate systems BTRK1 and BTRK2 in the localizercoordinate system LCLZ. Based on the same signals received for theinstrument tracker 58, the localization engine 100 determines the poseof the instrument tracker coordinate system TLTR in the localizercoordinate system LCLZ.

The localization engine 100 forwards the signals representative of theposes of trackers 54, 56, 58 to a coordinate transformer 102. Coordinatetransformer 102 is a software module that runs on navigation processor62. Coordinate transformer 102 references the data that defines therelationship between the pre-operative images and/or the virtual modelsof the patient and the bone trackers 54, 56. Coordinate transformer 102also stores the data indicating the pose of the treatment end of thesurgical instrument 30 relative to the instrument tracker 58. Coordinatetransformer 102 also references the data that defines the virtualobjects, if separate from the virtual models.

During the procedure, the coordinate transformer 102 receives the dataindicating the relative poses of the trackers 54, 56, 58 to thelocalizer 44. Based on these data and the previously loaded data, thecoordinate transformer 102 generates data indicating the relativeposition and orientation of both the coordinate system EAPP, and thebone coordinate systems, FBONE, TBONE to the localizer coordinate systemLCLZ.

As a result, coordinate transformer 102 generates data indicating theposition and orientation of the treatment end of the surgical instrument30 relative to the target sites against which the treatment end isapplied. Image signals representative of these data are forwarded todisplays 36, 38 enabling the surgeon and staff to view this information.In certain embodiments, other signals representative of these data canbe forwarded to the manipulator controller 64 to guide the manipulator66 and corresponding movement of the surgical instrument 30. Thus, thisdata also indicates a virtual location of the treatment end of thesurgical instrument 30, which may also be modeled as a separate virtualobject, with respect to the virtual models and the virtual objects.

Referring back to FIG. 1, the guidance station 26 further includes avision device 72. In the embodiment shown, the vision device is mountedto the localizer camera unit 46. In other embodiments, the vision device72 may be mounted on a separate adjustable arm to position the visiondevice 72 separately from the localizer camera unit 46. The visiondevice 72 is preferably placed with a field of view of the target sitesfree from obstructions. The vision device 72 has a vision controller 73in operative communication with the navigation computer 34. The visiondevice 72 may also be referred to as an imaging device or a digitalimaging device capable of capturing 3-D images in real-time. One exampleof a suitable vision device is the commercially available Kinect SDK orsimilar Kinect model, sold by Microsoft Corporation. In otherembodiments, the vision device 72 may comprise a laser array or a stereocamera system.

The vision device 72 has an outer housing 76 that supports one or moreimage sensors 78, 79. One of the image sensors may be a depth imagesensor 78 used to identify a depth image, while the other image sensormay be a color image sensor 79 used to generate color images. Both imagesensors 78, 79 may be in the form of CMOS sensors or other suitablesensors. Additionally, a light source 80 is supported in the housing 76to generate and transmit light that is reflected back by surfaces in thefield of view of the depth image sensor 78.

The sensors 78, 79 and the light source 80 communicate with the visioncontroller 73 to determine the distances of the surfaces in the field ofview with respect to a vision coordinate system VIS (see FIG. 3). In oneembodiment the light source 80 emits infrared light and the visioncontroller 73 determines the elapsed time required for the infraredlight to reflect off the surfaces in the field of view and return to thedepth image sensor 78. This process is repeated over a plurality ofiterations to determine distances from the vision device 72 to surfacesin the field of view of the vision device 72 so that a point cloud 202can be generated (see FIG. 4).

The navigation computer 34 communicates with the vision controller 73 toreceive signals and/or data representative of the point cloud 202.Imaging software, comprising an image generator module, is loaded on thenavigation computer 34 and run by the navigation processor 62 to createthe point cloud 202 based on the field of view of the vision device 72.The point cloud 202 is created in the vision coordinate system VIS. Thepoint cloud 202 is a set of image data points in the vision coordinatesystem VIS that correspond to the surfaces in the field of view of thevision device 72. These image data points are defined by x, y, zcoordinates. The point cloud 202 can be saved or stored as an image datafile.

It should be appreciated that by integrating the vision device 72 intothe localizer camera unit 46, the vision coordinate system VIS can beeasily registered to the localizer coordinate system LCLZ since thelocation of the image sensors 78, 79 relative to the optical sensors 50,and vice versa, is known and fixed. During manufacturing the visiondevice 72 can be calibrated to the localizer 44 to generate data withrespect to the same coordinate system so that the vision coordinatesystem VIS does not need to be transformed to the localizer coordinatesystem LCLZ via the coordinate transformer 102.

In other embodiments, such as those in which the vision device 72 isseparate from the localizer camera unit 46, the vision device 72 mayhave a tracker (not shown) rigidly mounted to the housing 76 toestablish a relationship between the vision coordinate system VIS andthe localizer coordinate system LCLZ. For instance, using preloaded datadefining a relationship between the tracker's coordinate system and thevision coordinate system VIS, the coordinate transformer 102, based onthe position of the tracker in the localizer coordinate system LCLZ,could transform the vision coordinate system VIS to the localizercoordinate system LCLZ.

Referring to FIG. 4, the vision device 72 collects images of the targetsites and the surfaces surrounding the target sites that are in thefield of view of the vision device 72. In the embodiment shown, thevision device 72 collects images of the target site TS and the surfacessurrounding the target site TS that are in the field of view of thevision device 72. The navigation computer 34 cooperates with the visioncontroller 73 to create the point cloud 202 of the target site TS andthe surfaces surrounding the target site TS, which defines image dataassociated with the target site TS and the surfaces surrounding thetarget site TS.

At the same time that the image data is being generated, the localizerdata is also being generated. The navigation computer 34 cooperates withthe localizer 44 to determine a position and orientation of the virtualmodels and the virtual objects defining the target sites in thelocalizer coordinate system LCLZ. In the embodiment shown, thenavigation computer 34 cooperates with the localizer 44 to determine aposition and orientation of the virtual model VM of the femur F and theposition and orientation of the virtual object VB in the localizercoordinate system LCLZ. This localizer data comprises the model datadefining the virtual model VM and the virtual object VB. In some cases,the model data includes data points in the form of a point cloudassociated with the virtual model VM and a separate point cloudassociated with the virtual object VB.

Still referring to FIG. 4, the navigation processor 62 runs a data mergemodule 101 (see FIG. 1), which is a software module that merges thelocalizer data and the image data to yield merged data (once thelocalizer data and the image data is located in, or transformed to, acommon coordinate system). The merged data represents a second virtualobject VR that defines a region R to be avoided during the surgery thatis outside of the target site TS. This merging of data is illustrated byarrows in FIG. 4. In the embodiment shown, the merged data thatrepresents the second virtual object VR may comprise: (1) data points204 associated with bone that is to be avoided by the surgicalinstrument 30 that is outside of the target site TS; (2) data points 206associated with exposed soft tissue that is to be avoided by thesurgical instrument 30 that is outside of the target site TS; (3) datapoints 208 associated with retractors that are to be avoided by thesurgical instrument 30; and (4) data points 210 associated with skin ofthe patient that is outside of the target site TS.

In some embodiments, like that shown in FIG. 4, the merged datacomprises all data points in the point cloud 202 that have coordinateslocated outside of the virtual object VB after the localizer data andthe image data are merged. In some cases, when a path for the treatmentend of the surgical instrument 30 to reach the target site TS is notcompletely clear, such as when the target site TS is at least partiallyobstructed by soft tissue or other sensitive anatomical structures,defining all visible surfaces outside of the target site TS as part ofthe second virtual object VR can be particularly advantageous so thatthe surgical instrument 30 is able to avoid any sensitive anatomicalstructures, tools, etc., that are located outside of the target site TS.

The merged data that represents the second virtual object VR, and whichdefines the region R to be avoided, can be processed by the navigationprocessor 62 so that a representation thereof can be displayed to theuser on the displays 38, 39 and the user can visualize a position andorientation of the surgical instrument 30 relative to the region R. Insome cases, the data points that virtually define the region R to beavoided can be converted into a mesh surface, a constructive solidgeometry (CSG), voxels, or other virtual object types using variousvirtual object representation techniques. Additionally, the navigationprocessor 62 may automatically limit the size of the second virtualobject VR, and thus the extent of the region R, to a predefined distancefrom the target site TS, or the user may be able to manually refine thesecond virtual object VR, including defining an outer perimeter of thesecond virtual object VR.

It should be noted that the second virtual object VR may change inconfiguration (e.g., size, shape, position, etc.) during the surgicalprocedure owing to the elastic and/or flexible nature of some of thetissues in the region R defined by the second virtual object VR.Additionally, the region R may change as retractors are adjusted, or asadditional tools or equipment are brought into and out of the field ofview of the vision device 72. In other words, the nature of the region Rto be avoided is dynamic and may continuously change, but with thenavigation techniques described herein, the second virtual object VR canbe continuously updated (e.g., at a predefined frequency) with each newset of image data and localizer data so that the user is able to avoidthe region R to be avoided during the surgical procedure regardless ofchanges to the region R.

The second virtual object VR that defines the region R to be avoided canalso be transmitted to the manipulator controller 64 and treated as a“no-fly” zone in which the treatment end of the surgical instrument 30is prevented from entering. As a result, when the manipulator 66operates in an autonomous mode, the manipulator 66 is able to controlpositioning of the surgical instrument 30 to avoid the region R andthereby avoid sensitive anatomical structures, such as soft tissue andbone to be preserved, and tools, such as retractors, suction tubes, andthe like, located near the target site TS.

Referring to FIG. 5, one embodiment of a method for determining theregion R to be avoided is shown. In step 300, a surgeon or other medicalprofessional creates a surgical plan for the patient. The surgical planidentifies the surgical procedure to be performed and the treatment tobe undertaken. The surgical plan is often based on pre-operative images,such as images taken from MRI or CT scans, which are converted into a3-D virtual model VM of the patient's anatomy. The virtual object VBdefining the target site TS to be treated during the surgical procedureis also generated and associated with the 3-D virtual model VM as partof the surgical plan.

In step 302, data relating to the virtual model VM and the virtualobject VB, which defines the target volume of material to be treated atthe target site TS, such as the target volume of bone to be removed, aretransferred to the navigation computer 34 to be stored in the navigationcomputer 34.

In step 304, localizer data is then generated. The localizer datacomprises data associated with the positions and orientations of thevirtual model VM and the virtual object VB in the localizer coordinatesystem LCLZ. Image data is simultaneously being generated in step 306 sothat at each time step during navigation, there is correspondinglocalizer data and image data. The image data comprises the point cloud202 which comprises the position and orientation of surfaces in thefield of view of the vision device 72, including surfaces of the targetsite TS and surfaces outside of the target site TS.

In step 308, the data merge module 101 of the navigation computer 34evaluates the localizer data and the image data. In particular, the datamerge module 101 merges data points from the image data (e.g., the pointcloud 202) with data points from the localizer data (e.g., data pointsfor the virtual object VB). In step 310, the data merge module 101 thenidentifies all of the data points from the image data that fall outsideof the virtual object VB. This remaining data set yields the region R tobe avoided, which is then saved in memory in the navigation computer 34as the second virtual object VR to be avoided by the surgical instrument30. In step 312, the user operates the surgical instrument 30, eithermanually, or robotically, to remove the target volume of tissue from thetarget site, while avoiding the region R. The steps 304-312 repeat foreach processing time step during navigation until the surgical procedureis complete, e.g., until all the tissue has been removed from the targetsite TS. As a result, the method is able to compensate for changes tothe region R during the surgical procedure.

In other embodiments, it should be appreciated that the systems andmethods described herein for merging localizer data and image data couldsimilarly be performed to generate other types of virtual objects, otherthan virtual objects that define regions to be avoided, like the regionR. For instance, the localizer data and the image data could be mergedto yield virtual objects that define target sites, such as volumes ofmaterial to be removed, desired trajectories for the surgical instrument30, and the like. Additionally, the image data and the localizer datacould be merged for other purposes.

As will be appreciated by one skilled in the art, aspects of the presentembodiments may take the form of a computer program product embodied inone or more computer readable medium(s) having computer readable programcode embodied thereon. Computer software including instructions or codefor performing the methodologies described herein, may be stored in oneor more of the associated memory devices (for example, ROM, fixed orremovable memory) and, when ready to be utilized, loaded in part or inwhole (for example, into RAM) and implemented by a CPU. Such softwarecould include, but is not limited to, firmware, resident software,microcode, and the like.

Several embodiments have been discussed in the foregoing description.However, the embodiments discussed herein are not intended to beexhaustive or limit the invention to any particular form. Theterminology which has been used is intended to be in the nature of wordsof description rather than of limitation. Many modifications andvariations are possible in light of the above teachings and theinvention may be practiced otherwise than as specifically described.

What is claimed is:
 1. A surgical navigation system for performingsurgery at a target site on a patient, the target site defined by avirtual object, said system comprising: a patient tracker configured tobe attached to the patient; a localizer configured to cooperate withsaid patient tracker to generate localizer data associated with thetarget site; a vision device configured to generate image dataassociated with the target site and surfaces surrounding the target siteduring the surgery; a navigation computer coupled to said localizer andsaid vision device to determine a region to be avoided outside of thetarget site during the surgery based on said localizer data and saidimage data.
 2. The system as set forth in claim 1 wherein saidnavigation computer is configured to determine said region to be avoidedby analyzing said localizer data and said image data and said navigationcomputer is configured to generate a second virtual object defining saidregion.
 3. The system as set forth in claim 2 wherein said navigationcomputer has a coordinate transformation module configured to combinesaid localizer data and said image data into a common coordinate system.4. The system as set forth in claim 3 wherein said navigation computerhas a data merge module configured to evaluate said image data and saidlocalizer data in said common coordinate system to determine the regionto be avoided by merging said image data and said localizer data to formmerged data and selecting at least a portion of the merged data todefine the region to be avoided which represents surfaces surroundingthe target site that are outside of the target site.
 5. The system asset forth in claim 4 wherein said image data generated by said visiondevice comprises a three dimensional map of at least a portion of thesurfaces surrounding the target site that are outside of the targetsite.
 6. The system as set forth in claim 5 wherein said threedimensional map comprises one or more of a point cloud, a range map, aplane, a line, or a single point.
 7. A robotic surgical system forperforming surgery at a target site on a patient, the target sitedefined by a virtual object, said system comprising: a robotic device;an end effector coupled to said robotic device and configured to treatthe patient at the target site; a patient tracker configured to beattached to the patient; a localizer configured to cooperate with saidpatient tracker to generate localizer data associated with the targetsite; a vision device configured to generate image data associated withthe target site and surfaces surrounding the target site during thesurgery; and a navigation computer coupled to said localizer and saidvision device and configured to determine a region to be avoided outsideof the target site during the surgery based on said localizer data andsaid image data, wherein said navigation computer is coupled to saidrobotic device so that said robotic device is operable to move said endeffector with respect to the target site while avoiding the region to beavoided.
 8. A method for performing surgery at a target site defined bya virtual object, utilizing a patient tracker attached to a patient, alocalizer, a vision device, and a navigation computer, said methodcomprising the steps of: generating localizer data associated with thetarget site while the patient tracker is attached to the patient;generating image data associated with the target site and surfacessurrounding the target site; and determining a region to be avoidedoutside of the target site during the surgery based on the localizerdata and the image data.
 9. The method as set forth in claim 8, whereindetermining the region to be avoided outside of the target site duringthe surgery based on the localizer data and the image data comprisesanalyzing the localizer data and the image data to generate a secondvirtual object defining the region.
 10. The method as set forth in claim9, wherein determining the region to be avoided outside of the targetsite during the surgery based on the localizer data and the image datacomprises combining the localizer data and the image data into a commoncoordinate system.
 11. The method as set forth in claim 10, whereindetermining the region to be avoided outside of the target site duringthe surgery based on the localizer data and the image data comprises:evaluating the image data and the localizer data in the commoncoordinate system; merging the image data and the localizer data to formmerged data; and selecting at least a portion of the merged data todefine the region to be avoided which represents surfaces surroundingthe target site that are outside of the target site.
 12. The method asset forth in claim 11, wherein generating the image data associated withthe target site and surfaces surrounding the target site comprisesgenerating a three dimensional map of at least a portion of the surfacessurrounding the target site that are outside of the target site.
 13. Themethod as set forth in claim 12, wherein generating the threedimensional map of the at least a portion of the surfaces surroundingthe target site that are outside of the target site comprises generatingone or more of a point cloud, a range map, a plane, a line, or a singlepoint.
 14. A method for performing robotic surgery at a target sitedefined by a virtual object, utilizing a patient tracker attached to apatient, a localizer, a vision device, a navigation computer, and an endeffector attached to a robotic device, said method comprising the stepsof: generating localizer data associated with the target site while thepatient tracker is attached to the patient; generating image dataassociated with the target site and surfaces surrounding the targetsite; determining a region to be avoided outside of the target siteduring the surgery based on the localizer data and the image data; andtreating the patient while avoiding the region to be avoided.